ExPLAIND: Unifying Model, Data, and Training Attribution to Study Model Behavior

Post-hoc interpretability methods typically attribute a model’s behavior to its components, data, or training trajectory in isolation, and are often tied to a particular level of granularity along the local-to-global spectrum. This leads to explanations that lack a unified view and may miss key interactions. We present ExPLAIND, a theoretically grounded, unified framework that integrates model components, data, and training trajectory while supporting explanations across granularities. We generalize recent work on gradient path kernels, reformulating models trained by AdamW as kernel machines. From the resulting kernel feature maps, we derive novel parameter-wise and step-wise influence scores. We empirically validate the resulting decomposition of model behavior in several settings and apply ExPLAIND to two case studies. Our findings on a Transformer exhibiting Grokking support previously proposed learning phases, while refining the final phase as one in which outer layers align around a representation pipeline learned after memorization. For EuroLLM pretraining, ExPLAIND reveals a two-phase dynamic, with the first characterized by outer-layer MLP learning and the second by increased relative influence of intermediate attention layers. These results establish ExPLAIND as a unified framework for interpreting model behavior and training dynamics.

Modern artificial intelligence models can give impressive answers, but we still do not know enough about how those answers are formed. Existing methods often look at only one part of the story at a time. For example, one method may focus on which parts of a model matter most, while another may focus on which examples from the data shaped the model most strongly. For each of these pquestions, we have separate tools to study these influences. To get the complete picture, however, one must piece together several separate methods and important interactions can be missed. We created ExPLAIND, a new method that combines those views into one framework. We designed ExPLAIND to show how different parts of a model work together and how their roles change as the model is learning from data. We tested ExPLAIND in different experiments on both small and large models, including a recent open-source large language model. In those experiments, ExPLAIND revealed new patterns in how these systems develop. ExPLAIND and our results help move the field toward a better understanding of what happens inside these models. This is important for two reasons: First, it enables guiding future improvements in machine learning. Second, it helps make AI systems more transparent enabling mechanisms that ensure we implement them safely into our society. Second, it can help make artificial intelligence systems more transparent, which supports ongoing legal and technical efforts to make them safer.